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Abstract 

It can be difficult to foster focused and effective communication in online discussions within 
large classes. Implementing protocols is a strategy that may help students communicate more 
effectively, facilitate their learning process, and improve the quality of their work within online 
discussions. In this exploratory research study, a protocol was developed and improved over two 
iterations in a very large undergraduate video-streaming business course (M=412; A2=450). The 
discussion instructions were consolidated and adjusted, and design elements such as a grading 
rubric, exemplary student samples, and due date reminders were added in the second iteration. 
There were higher perceptions of social, cognitive, and teaching presences in the second 
iteration, as well as significantly more group cognition within the discussion measured through a 
Community of Inquiry coding template. Findings suggest that protocols are a potentially useful 
strategy to manage online discussions in large classes. 
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Introduction 

Asynchronous discussions play an important role in online learning by providing a space 
for instructors and students to fonn a community, to engage in dialogue about the course content, 
and to co-construct knowledge (Gao, Zhang, & Franklin, 2013). Because of the asynchronous 
nature of the discussions, participants have more time to think before responding, and the act of 
writing elicits the fonnation of new knowledge and ideas (Hew, Cheung & Ng, 2010). In 


Online Learning - Volume 21 Issue 1 - March 2017 


165 



addition, discussion forums provide a pennanent space for the participants to return to their 
original contributions, promoting reflection (Hew et ah, 2010) and self-assessment (Gao et al, 
2013). 


Despite these affordances, it can be challenging to create and sustain focused, in-depth 
online discussions (Gao et ah, 2013). In a literature review, Hew et al. (2010) identified several 
reasons why students do not fully participate in online discussions. Students will not fully 
participate when they do not see the purpose of the discussion, do not understand what to 
contribute, do not receive responses to their posting, and cannot make sense of the discussion 
due to the structure of the online forum. Therefore, online discussions must be structured in a 
way that clearly communicates their purpose and student expectations, encourages students to 
co-construct knowledge, and facilitates meaningful discussion. Past research has specifically 
identified the establishment of clear communication protocols and requirements for participating 
as vital for a successful discussion (Brannon & Essex, 2001; Darabi, Liang, Survavanshi, & 
Yurekli, 2013; Makitalo, Weinberger, Hakkinen, Jarvela, & Fischer, 2005). An increasingly 
common constraint, however, is class size. While an optimal online class size is between 12 to 
16 students for effective communication (Orellana, 2006; Tomei, 2006), online courses can 
sometimes hold hundreds of students. Even when sectioned into groups, it can be difficult for 
instructors to effectively facilitate online discussions due to logistical and grading issues. 

The purpose of this paper is to explore the use of a discussion strategy called a protocol, 
which may potentially address the challenge of stimulating a productive discussion in a large 
online class. Protocols explicitly structure meaningful conversations with the purpose of 
stimulating student problem solving, reflection, and elicitation of support from others 
(McDonald, Zydney, Dichter, & McDonald, 2012). Protocols establish a well-defined goal, clear 
roles, and set rules for interactions, and specific deadlines for posting. A notable example of a 
protocol is called the Tuning protocol (McDonald et al., 2012). The goal of this protocol is to 
improve (“tune”) a particular work in progress. For this protocol, a participant shares the work in 
progress and has the opportunity to ask the others for relevant feedback. Others listen about the 
work in progress, and then ask clarifying and probing questions before supplying focused 
feedback. Finally, the participant reflects on the feedback and brainstonns ways to improve the 
work. This participant also listens and provides feedback to the others. In this way, protocols 
sharpen communication, enhance collective thinking, and build knowledge. The Tuning protocol 
exhibits the four core characteristics of a protocol that set it apart from other structured 
approaches (McDonald et al., 2012). First, all participants have a voice and play a designated 
role. Second, all participants engage in different and varied ways. Third, all participants have the 
dual roles of both creating as well as reading text. Finally, protocols foster trust since the norms 
are well established. 

Protocols first started within face-to-face environments, but are now being explored in 
the online environment (McDonald et al., 2012). This study explored the use of protocols within 
online discussions in a large enrollment college course over a two-semester period. The explicit 
structure of a protocol may keep students in a large online class on track and focused on a goal, 
sustain the community, and eliminate the need for constant facilitation from the instructor. At the 
conclusion of the paper, specific enhancements that enable a protocol-based discussion to 
logistically work in a large online class are proposed. 
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Theoretical Background: Community of Inquiry 

The Community of Inquiry (Col) framework has been used as one of the premier theories 
in the last two decades to conceptualize community in many online discussion research studies 
(see Col website ). The framework proposes three essential presences that contribute to a 
successful educational experience: social, cognitive, and teaching (Garrison, Anderson & Archer, 
2000; Garrison, Anderson, & Archer, 2010). Social presence is the ability of learners to project 
themselves socially and emotionally, being perceived as “real” people in mediated 
communication (Garrison & Arbaugh, 2007; Rourke, Anderson, Garrison & Archer, 1999). 
Cognitive presence refers to the extent to which learners are able to construct and confinn 
meaning through sustained reflection and discourse (Garrison, Anderson & Archer, 2001 & 
Garrison & Arbaugh, 2007). Finally, teaching presence is defined as the “design, facilitation, and 
direction of cognitive and social processes for the purpose of realizing personally meaningful 
and educationally worthwhile learning outcomes” (Anderson, Rourke, Garrison, & Archer, 2001, 
p. 5). 


Using these presences as a framework, an effective online discussion will encourage 
students to not only communicate on a social level but also engage in an academic conversation, 
while being purposely directed to achieve learning outcomes. 

Literature Review 


Class Size 

There is no standardized definition of what constitutes a “large class.” Maringe and Sing 
(2014) define large class size as “any class where the numbers of students pose both perceived 
and real challenges in the delivery of quality and equal learning opportunities to all students, in 
that classroom” (p. 763). The researchers adopt Maringe and Sing’s definition for purposes of 
this paper and regard our target classes of over 400 students as large. 

In the literature, class size was found to be a factor in students’ interactions in online 
courses (Hewitt & Brett, 2007; Orellana, 2006). The larger the class size is, the less likely 
students will actively engage in online activities (Rocca, 2010). Consequently, class size also 
negatively affects students’ social engagement as well as creates infonnation overload for 
students (Hewitt & Brett, 2007). Past research has found that to maintain sufficient instructor- 
student interactions, the ideal online class size was between 12 to 16 students to achieve the 
highest level of interactions (Orellana, 2006; Tomei, 2006). Due to student enrollment and 
administrative pressure, however, class sizes have continued to increase in American colleges in 
recent years. In this study’s school, a regular general education course can be over one hundred 
students. 

A search via Google Scholar and the Education Research Infonnation Center (ERIC) 
database using the keywords “large class” and “online discussion” was conducted. 
Unfortunately, only a few journal articles were found on the topic of facilitating discussions in 
large online classes (see Yang, 2007). For large classes, it is recommended to section students 
into small groups to achieve high quality interactivity (Kim, 2013). Hew and Cheung (2011) 
found that as discussion group size increased up to 10 students, the level of knowledge 
construction also increased. Although prior research does identify strategies that are effective in 
eliciting knowledge in online discussions (see deNoyelles, Zydney, & Chen, 2014), these studies 
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often do not sample large enrollment online classes. Thus, more research is needed to detennine 
how to meet the challenge instructors face in creating an effective online discussion activity in 
large undergraduate classes. 

Online Discussions and Protocol-based Discussions 

Research about online discussions in large classes is rare (Eyitayo, 2005). One study 
found that small group, online discussions can help students develop their critical thinking skills 
with skillful facilitation in a class of 133 students (Yang, 2007). The instructors divided the class 
into smaller groups and assigned live teaching assistants to be group facilitators who used 
Socratic questions. Unfortunately, most instructors who teach large classes are not equipped with 
multiple teaching assistants to assist with discussion facilitation. Given the constraints, is it still 
possible to use online discussions as a teaching/engagement strategy in large online classes? 

In a review of literature, deNoyelles, Zydney & Chen (2014) identified discussion 
strategies associated with higher levels of the Col presences. One effective strategy noted was 
the use of protocols; however, empirical research on protocols in online discussions is scarce. 
One study that compared a protocol and a non-protocol online discussion in two small sections 
of a graduate course found that the three Col presences exhibited by students were more evenly 
distributed in protocol-based discussions (Zydney, deNoyelles & Seo, 2012). The protocol 
promoted more shared group cognition, rather than individual monologues. It also significantly 
increased students' opportunity to participate in the instructional design of the course, making 
teaching presence a shared responsibility between teachers and students. The findings indicated 
that the shared responsibility reduced the burden on the instructor for doing all the facilitation 
and enabled the instructor more time to diagnose misconceptions and inject knowledge when 
necessary. 

Skillful facilitation is especially difficult in large online classes. Considering the benefits 
discovered from prior research on protocols, the researchers expected that protocol-based 
discussions could benefit large online classes where extensive instructor facilitation is not 
feasible. Therefore, our research focused on how to utilize one specific discussion strategy, the 
protocol, in order to improve students’ perception and enactment of Col in online discussions 
within large classes. 


Methodology 

This study examined what enhancements are needed to enable a protocol-based 
discussion to logistically work in a large class, and whether the enhancements made in the 
second design iteration of the protocol-based online discussions improved the Col for a large 
enrollment course. Two research questions were posed: 

(1) How did students in a large enrollment online course perceive the enhancements 
made to the protocol-based discussions? 

(2) How did the enhancements made to the protocol-based discussions impact the 
elements that contribute to a Col? 
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The Context of the Study 

Online discussion protocols were developed and implemented in a large undergraduate 
business course (GEB3113) over two semesters (Iteration 1 and 2) in 2014, taught by the same 
instructor at the University of Central Florida. The course was classified as “video streaming,” 
meaning that face-to-face attendance was optional, with the sessions being streamed to an online 
audience. Iteration 1 included 412 participants and Iteration 2 included 450. In both iterations, 
students regularly engaged in graded online discussions in groups of approximately 10 students 
(called “Bazinga circles”), resulting in at least 40 groups per semester. 

Three protocol-based discussions were implemented in the course, each focused on 
written class assignments, with the goal of encouraging students to reflect and give meaningful 
feedback about each other’s work. The protocol featured in this study was adapted from the 
Tuning protocol described in the introduction (McDonald et ah, 2012), and was the first one 
offered in the course. The piece of work in question was the business model, which is essentially 
a description of an innovative business concept. Included in the business model are elements 
such as customer relationships, revenue streams, and key resources. 

For both iterations, the online discussion protocol was divided into three basic parts: 

• Part A: Students posted their written assignment (business model), and then asked 
group members to consider at least one aspect in which they desired constructive 
feedback. 

• Part B: One week after Part A, students chose at least one person to whom to 
reply and provide feedback. 

• Part C: One week after Part B, students reflected on the next steps for further 
developing their business model based on peer feedback. 

• Once the discussion concluded, the students submitted the business model as a 
formal assigmnent for grading. 

Based on feedback from the students and instructor, problem areas with the discussion 
were identified and enhancements were made to the online discussion protocol used, resulting in 
Iteration 2 of the discussion protocol (Appendix A). 

Seven improvements were made after the first design iteration: 1) The discussion 
structure was simplified. In Iteration 1, Part A was in one discussion forum (i.e., a discussion 
board page), while Parts B and C were in a separate forum. As this resulted in confusion about 
how to interact with each other, in Iteration 2, Parts A and B were consolidated into one forum to 
encourage smoother peer feedback, with Part C (personal reflection) submitted as part of the 
final written Business Model assignment. 2) The feedback prompts were simplified. In Iteration 
1, students were told in part B to specify their peer feedback to be “warm” (describe what is 
working with the model), “cool” (consider aspects of work that need improvement) and/or 
“hard” (ask deeper questions that get at the larger aspects of the work). In an effort to simplify 
the instructions, in Iteration 2, this classification was omitted and students were only asked to 
provide feedback based on the first person’s post. Instead of including complicated directions, 
the warm, cool, and hard feedback was modeled in example discussion posts. 3) To simplify the 
focus of the discussion, the discussion and the Business Model assignment instructions were 
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separated. 4) Due dates for the multiple parts of the discussion were added to the course calendar 
in Iteration 2 to provide reminders to students. 5) Exemplary discussion samples were provided 
from past semesters in order to display good examples. 6) A rubric was added to explain how the 
discussion would be graded. 7) The point-value of the discussion went from 10 points to 30 
points in order to encourage more careful work. 

Data Collection and Preparation 

With the approval from the Institutional Research Board, the research team collected two 
types of data, survey and discussion posts, over the two iterations (Table 1) in the spring and fall 
semesters of 2014. Based on the Col framework, researchers created a survey instrument 
(Arbaugh et al., 2008) to measure the quality of online teaching and a coding template (Garrison 
et al., 2000) to analyze indicators of cognitive presence, social presence, and teaching presence 
in students’ discussion activities. The researchers used these two instruments in our research to 
evaluate students’ perceptions and discussions. At the end of each semester, a survey was 
distributed that asked students to assess the protocol discussions used in the course. This survey 
included two parts: a quantitative section that utilized the Col instrument (Appendix B), which 
assessed students’ perceived levels of social, cognitive and teaching presences with reference to 
the protocol-based discussions, and a qualitative section that included open-ended questions to 
assess students' feedback on the discussions. In addition to this survey, students’ discussions 
were analyzed for the social, cognitive, and teaching presences. The triangulation of this data 
enabled us to examine Col from two main perspectives: the closed-ended questions were 
directed mainly at the instructor role in fostering a Col, the coding of the group discussions was 
focused on the students' contribution to Col in the discussions, and the open-ended questions 
reflected both the instructor and student involvement. 

Table 1 

Data Source Types, Analyses, and Participant Number _ 


Data Source Analysis Iteration 1 Iteration 2 


Semester 


Spring 

Fall 

Survey Closed-ended 

Descriptive & 
Independent 
Samples T-Test 

N= 394 

A=446 

Survey Open-ended I 

Content Analysis 

N= 394 

A=446 

Survey Open-ended II 

Content Analysis 

N= 394 

A=446 

Discussion posts 

Content Analysis 
& 

Fisher’s Exact Test 

10 (One selected 
group) 

11 (One selected 
group) 
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Analyses 

For Research Question 1, the research team analyzed the open-ended survey results to 
evaluate students’ perceptions towards the enhancements made to the protocol-based 
discussions. For Research Question 2, the researchers analyzed both the quantitative survey 
results and the discussion posts to assess if the enhancements made to the protocol-based 
discussions impacted the Col elements. 

Open-ended survey analysis. Students in both iterations were given open-ended 
questions as part of the course evaluation. In total, 840 students responded with comments on the 
protocol-based discussions. One open-ended question asked them to provide feedback about the 
discussions. In Iteration 2, an additional question asked students how the discussion contributed 
to their perfonnance on the written assignment. A conventional content analysis was applied to 
the first open-ended survey question. Conventional content analysis emphasizes becoming 
immersed in the data so that new insights can emerge (Hsieh & Shannon, 2005). For the initial 
analysis of the first open-ended survey question, one member of the research team who was 
unfamiliar with the course design of either iteration independently open-coded to reveal 
emerging themes, noting anything in relation to the main research questions. Examples of initial 
codes generated include group size, group composure, clarity of instructions, amount of 
feedback, type of feedback, and usefulness of feedback, among others. Codes were then 
organized into the following categories: instructions, group fonnation, improving learning, 
examples, and feedback. The second open-ended survey question was then analyzed, with the 
team member continuing with open coding, and filing them under the initial codes and categories 
when appropriate. Examples of emerging themes include clarity of instructions and purposeful 
group fonnation. Finally, a comparative qualitative analysis between the two iterations was 
perfonned on students’ responses. The differences among the themes between the two iterations 
were then highlighted. In addition to allowing the themes to emerge from the data, the research 
team also discussed the themes in connection with the three presences of the Col framework 
within the Discussion section (Merriam, 2001). 

Quantitative survey analysis. For the quantitative survey results, frequencies were used 
to analyze infonnation regarding the level of presences. Independent samples t-tests were 
conducted to investigate the differences in perceptions between two iterations. 

Discussion post analysis. An in-depth analysis of the discussions of one group from 
Iteration 1 and 2 was conducted. The discussion posts were downloaded, cleared of student 
names and identifying information, and stored on a secure server as randomly numbered groups. 
To prepare for the coding process, one member of the research team with expertise in the 
modified Col coding system (see Table 2) devised in an earlier study (Zydney et al., 2012) 
trained two other members. The main difference between the original Col coding system 
developed by Garrison et al. (2000) and the modified one is that it separated the “individual and 
group categories within cognitive presence to distinguish between participants answering 
questions as a monologue unrelated to other participants’ postings versus participants 
interactively discussing the topic with one another” (Zydney et al., 2012, p. 81). For this 
process, each complete thought was the unit of analysis and was coded for cognitive, social, and 
teaching presence as defined by this model. Each unit of analysis was coded for the highest level 
of cognitive presence and could have multiple social or teaching indicators. To establish the 
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coding process among the research team, a practice group of posts was selected and coded by 
two members of the research team. The third observed to understand the process and provide 
input on any cases about which the original coders did not agree. A random group of posts from 
each iteration was then coded independently by two researchers who later met to resolve 
discrepancies. A total of 366 ratings was assigned with 47 discrepancies for an inter-rater 
reliability of 87%. All discrepancies were resolved for a 100% agreement. 

After the discussion posts were coded, frequencies and percentages for each category by 
individual and group presence were calculated to analyze the differences in the group between 
the two iterations. In addition, the Fisher’s Exact test (2-sided) was used to identify statistically 
significant results. This nonparametric test was chosen because of the small sample size. 
Following this analysis, further qualitative investigations were done to identify differences 
between protocol iterations. Examples of student posts were used to illustrate any variances. 

Table 2. 

Col Codins Template (adapted from Garrison et al. (2000)) _ 

Elements Categories Indicators Example 


Cognitive Triggering Event 
Presence 

Exploration - 
Individual 


Exploration - 
Group 


Integration - 
Individual 


Integration - 
Group 


A new or related topic is raised 

Ideas, experiences, and prior 
knowledge are repeated and/or 
described 


Ideas are exchanged among the 
group, questions are asked to 
confirm understanding 


Prior knowledge and/or 
experience is related to the 
text; texts are related to each 
other 


Ideas of others are connected 
and/or expanded upon; 
counterpoints raised; 
improvements are suggested 


NA a 

“Obstacles I am anticipating 
include funding the business and 
establishing brand awareness and 
loyalty.” 

“I like your idea to make a 
Bookstore/Cyber Cafe that would 
sell college books under retail 
value.” 

“I’ve talked to many friends and 
family members which has caused 
me to want my product to be a 
value based product instead of 
cost driven." 

“While I agree with his premise 
that word of mouth advertising is 
great, it’s not the least bit practical 
for a business that’s just starting 
out; you must advertise through 
other mediums first to get your 
name out there.” 
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Social 

Presence 


Teaching 

Presence 


Resolution - 
Individual 


Applying ideas to personal 
future, theory to practice 


“Many large groups have the pain 
of paying far too much for too 
many rooms to fit all of their 
members in... My hotel chain will 
solve these problems by... putting 
the large groups into larger rooms 
for far less cost to them.” 


Resolution - Applying what has been “The response I got from X 

Group discussed completely turned my original 

idea around. At first, my plan was 
having the customers get picked 
up from home by a professional 
driver who takes them for a ride. 
This idea proved to be risky 
because it would be incredibly 
tricky to get an insurance 
company to cover injuries that can 
possibly occur on public roads. 
With having a closed course, I can 
stay within certain parameters to 
severely lessen liability on my 
business.” 


Emotional 

Expression 


Including emoticons, humor, 
and statements which exhibit 
emotion 


“Feel free to give me a ring when 
and if you need any graphic 
design or photography work done, 
D, ha-ha.” 


Open Complimenting someone, “XX, I think the concept of 

Communication expressing oneself freely, bringing the gym to the customer 

sharing stories, including is a great idea.” 

names 


Group Cohesion Encouraging collaboration, “I hope this helped a little!” 

community building, helping 
and supporting 


Instructional Identify and prompt discussion NA a 

design and topics 

organization 
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Facilitating 

discourse 


Asking others for feedback, 
encouraging the direction of 
the discussion 


“If anyone has any feedback or 
ideas they would like to share on 
reducing costs, I would greatly 
appreciate it!” 


Direct instruction Explain content in an NA a 

authoritative way, provide 
support for claims, providing 
resources 


a NA - not applicable because it did not show up in the groups’ discussions. 


Results 

A number of themes developed for the first open-ended survey responses including the 
clarity of instructions, purposeful group formation, improving learning and gaining new 
perspectives, encouraging community and collaboration, providing examples and eliciting 
effective feedback. An overall comparison of open-ended responses between iterations yielded a 
variety of similarities and differences among the six themes. 

Similar themes between iterations. Overall, responses across iterations were found to 
be similar for clarity of instructions, purposeful group fonnation, improving learning and gaining 
new perspectives and encouraging community and collaboration. 

Clarity of instructions. The responses concerning the instructions and access to the 
discussions were overwhelmingly positive with almost all students reporting clear, easy-to-use 
instructions as one student illustrated, “The discussions were very accessible and easy to reply to 
other posts. I thought the instructions were spot on and I had no difficulties.” A very small 
amount of the responses suggested a variety of changes. A few students commented that the 
instructions were too long, with one student suggesting that, “The instructions were too complex 
that it caused confusion... If the instructions were simplified then they would be easier to 
follow.” A few students suggested that it was initially unclear where or when to post responses, 
or the proper place was difficult to find among all the other groups. A small number of students 
also requested reminders of dates that responses were due, or suggested that some due dates were 
unclear. One student suggested that, “I believe the instructions could have been described in a 
way for it to be easier to understand along with the due dates." 

Purposeful group formation. Many suggestions focused on group fonnation, however, 
the suggestions varied greatly. Some students wanted a larger group for more feedback, while 
others asked for smaller groups for more personal communication. For example, one student 
suggested that future discussions should include “larger Bazinga circles [discussion groups] to 
ensure more replies and comments being shared.” In contrast, another student suggested that, 
“maybe by making groups smaller you could improve this experience.” Some students chose to 
focus on group composition by suggesting that groups include students of similar interests, while 
other suggestions asked for groups to include students of varied interest. To illustrate, one 
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student wrote, “I would have liked Bazinga groups to be picked because of similarities in the 
field even if it fell down to just separating it into products and services. Or what the industry is.” 

Improving learning and gaining new perspectives. Students also reported that their 
perfonnances on the class assigmnents were improved because the structured discussion allowed 
them to get multiple perspectives, iron out details, and identify problems. For example, one 
student commented that, “The structured discussions were very easy to understand and helped 
me learn more about my ideas from different perspectives.” In addition, giving feedback to other 
students was reportedly helpful by providing peer examples to compare to their own work. 
Overall, students felt that the protocol discussion helped them receive a better grade on the final 
project. One student summarized this idea by writing, “I really liked the discussions. This 
helped wrap everything up that the research team had learned. The end discussions, which I felt 
I was very prepared for, was thanks to the Bazinga discussions.” 

Encouraging community and collaboration. Finally, students reported that the group 
discussion made them feel better about their work and interacting with the class. One student 
reported that “the discussions gave me a feeling of belonging and collaborating with other 
students,” while another commented that, “I was able to interact with my peers more than in any 
other online course I’ve taken.” With class sizes in the 350-450 range, it is easy for students to 
feel disconnected, especially in an online setting. However, the protocol discussions may have 
helped students feel connected, with one student stating “I found the discussions helpful! They 
definitely helped me develop a sense of community within the class.” The protocol discussions 
were also reported to increase collaboration. As one student wrote, “I thought the discussions 
were great and made me feel like I was actually a part of the class. Being in such large classes is 
tough and having a small group collaborations helped immensely.” 

Different themes between iterations. Responses concerning provided examples and 
effective feedback were considerably different between groups. 

Providing examples. Although the overall response to clarity of instructions was similar 
between iterations, one difference emerged. Students in both iterations consistently responded 
that instructions were clear and easy to follow. Additionally, the students in Iteration 2 often 
elaborated that the provided examples were helpful. One student suggested 

The instructions were very clear and concise. I love that you guys give us clear 
instructions AND an example of what you are somewhat looking for. Makes working on 
the assigmnent a lot easier and is very reassuring to know I am on the right track. 

Eliciting effective feedback. Responses concerning effective feedback were considerably 
different between groups. The students in Iteration 2 more often offered positive responses 
regarding effective feedback, while the students in Iteration 1 compiled more negative responses 
in this area. Feedback from students in Iteration 1 more often identified the negative aspects of 
feedback including lack of feedback, ineffective last-minute feedback, or feedback that was 
minimal. For example, a student from Iteration 1 responded that 

I felt as though I didn’t receive adequate feedback through the circle discussions, and I 

never met any of the members in my group. I posted several assigmnents and asked for 


Online Learning - Volume 21 Issue 1 - March 2017 


175 



feedback but didn’t receive any. It was clear from the feedback that I did receive that 
some students did actually take the assignment seriously and attempt to provide me with 
a different perspective to aid me, but others were short and frankly quite useless. 

Students in Iteration 2 more often responded that the feedback was helpful and provided 
insight. As one student from Iteration 2 responded, “By using feedback from my Bazinga group 
members I was better able to come up with and expand on my business concept after taking into 
account the various ideas and contributions that my group members gave me. Very helpful!” 

Additional themes related to additional question for iteration 2. In Iteration 2, an 
additional question was added to the course survey to provide further insight into how the 
protocol-based discussions influenced the perfonnance on the Business Model assigmnent. A 
number of additional themes developed including guiding student progress, utilizing peer 
examples, and revising and refining ideas. 

Guiding student progress. Students felt that the protocol discussion helped them to keep 
pace with the course schedule and assignments. Students reported being forced to start 
assigmnents early, to understand material in order to be able to participate in discussions, and to 
keep up to date with responses. As a result, students had more time to reflect and modify their 
work before turning in the final copy. One student wrote, “The structured (protocol) discussions, 
like the Business Model assignment, helped me with time management. Since assignments were 
due first in the discussion it allowed me to schedule my time more efficiently.” 

Utilizing peer examples. In addition, students felt that the discussions allowed them to 
see examples of other student’s work, and provided a basis to compare their own work. As one 
student suggested, “I loved the structured (protocol) discussions! It was great to see 
many examples of what my peers were doing, to help improve my own understanding 
of the assignement and better my perfonnanace." 

Revising and refining ideas. The students reported that the protocol discussions helped 
to improve perfonnance by allowing them to revise and refine their ideas. The discussions 
allowed students to share their ideas, receive feedback from their peers, and adjust the 
assigmnents before submission. For example, a student responded 

They helped me in a number of ways, but the most is when I was missing something or 
didn’t go into detail about something, the people in my Bazinga group caught it and 
helped me score very high on these assignments. They would also tell me things like ‘I 
love your idea but have you thought about adding this?’ and with things like that it really 
helped me to refine my idea and made it stronger than I ever thought possible! 

Research Question 2: How Did the Enhancements Made to the Protocol-Based Discussions 
Impact the Elements that Contribute to a Col? 

A summary of the quantitative survey results and the quantitative and qualitative analysis 
of the discussion posts indicate that the enhancements made to the protocol-based discussions 
positively influenced elements of the Col. In particular, the cognitive presence among students 
was significantly higher in Iteration 2 than in Iteration 1. 
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Summary of quantitative survey results. Exploratory analyses (Table 3) showed that 
for both iterations, all three presences of the Col were perceived at high levels. However, 
students’ perceptions of teaching, social, and cognitive presences were significantly higher in 
Iteration 2. 


Table 3. 

Means and Standard Deviations of the Three Presences in the Two Iterations 
Iteration 1 (Spring 2014) Iteration 2 (Fall 2014) 



N 

Mean 

Std. Deviation 

N 

Mean 

Std. Deviation 

Teaching 

394 

4.04 

0.60 

446 

4.27 

0.51 

Presence 







Social Presence 

394 

3.79 

0.72 

446 

4.10 

0.63 

Cognitive 

394 

3.74 

0.76 

446 

4.07 

0.64 


Presence 


There were statistically significant differences in all three presences between the two 
iterations (Table 4). These results suggest that the improved discussion protocol in Iteration 2 did 
have an effect on students’ perception of online community. Specifically, when the discussion 
protocol is enhanced, students perceived a higher sense of teaching presence, social presence and 
cognitive presence. 

Table 4. 

Mean Differences and T-Value of the Three Presences Between the Two Iterations _ 


Mean Difference t df Sig. (2-tailed) 


Teaching 

.23 

6.05 

770 

<ooi*** 

Presence 





Social Presence 

.31 

6.60 

788 

<001*** 

Cognitive 

.33 

6.79 

773 

<001*** 

Presence 






Note: ***/?<.001. 


Discussion post analysis. Overall, the students’ discussions from Iteration 2 presented a higher 
frequency of social presence than the discussions from Iteration 1, as shown in Table 5. The 
frequencies and percentages of cognitive, social, and teaching presence in the discussions were 
similar between iterations. Further analysis was done to assess differences between iterations 
within each category. 
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Table 5. 

Overall Differences between Cognitive, Social and Teaching Presence 


Element 

n 

Iteration 1 


n 

Iteration 2 




Frequency 

% 


Frequency 

% 

Cognitive 

10 

48 

62 

9 

49 

55 

Social 

10 

21 

27 

9 

31 

35 

Teaching 

10 

8 

11 

9 

9 

10 

Total 

10 

77 

100 

9 

89 

100 


Cognitive presence. An examination of the instances of cognitive presence between 
iterations revealed a significantly higher rate of individual exploration in Iteration 1 in the 
discussions (p = .03) as shown in Table 6. In addition, this led to a significantly higher presence 
of total individual cognition for the students in Iteration 1 (p = .03). Subsequently, the students 
in Iteration 2 produced a significantly higher rate of total group cognition (p = .03) with higher 
frequencies achieved in all three categories. Overall, the students in Iteration 2 displayed a more 
even distribution between individual and group cognition. 

A qualitative examination of the discussion posts revealed further differences among the 
groups. Posts to part A of the discussion, in which students presented their business models and 
asked for feedback were similar across iterations. However, responses to parts B (providing peer 
feedback) and C (reflection and improvement of the business model) were shorter and coded at a 
lower cognitive presence in Iteration 1. For example, a typical response by the students in 
Iteration 1 to part B of the assigmnent was “It all looks pretty solid to me #4 good work.” 

While one of the shorter responses by a student in Iteration 2 was: 

Hey #7, love the idea! 

Many people in your target market are going to be interested in the business you have to 
offer and are going to love the service you provide them. Many large parties don’t like to 
be separated when having a good time in a hotel and hate to pay more just to get extra 
rooms. I like the idea of the unlimited access to entertainment in the lobby as well and 
the personal caterer. 

Your strategic alliance will be key to your business to get up and get going. Your 
business is going to need key investors that are willing to work with you and believe in 
the work you’re going to get done. Liability may be an issue as well because when you 
have a lot of people in one room it can tend to get a bit reckless and things may break in 
the room that were not your businesses fault. So make sure you have liability covered 
when people check in for your services. 

In many cases, the responses of students from Iteration 1 to parts B and C of the assignment were 
1 to 2 sentences that were coded with low cognitive presence or with none at all. However, the 
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students in Iteration 2 more often responded in multiple paragraphs that were coded at various 
group levels. 


Table 6 

Differences in Categories within Cognitive Presence 


Element 

Iteration 1 


Iteration 2 


P 


Frequency 

% 

Frequency 

% 


Triggering Event 

0 

0 

0 

0 

0 

Individual 






Exploration 

25 

52 

14 

29 

.03* 

Integration 

7 

15 

5 

10 

.55 

Resolution 

3 

6 

5 

10 

.71 

Sub-total 

35 

73 

24 

49 

.03* 

Group 






Exploration 

6 

10 

9 

18 

.58 

Integration 

7 

17 

14 

29 

.15 

Resolution 

0 

0 

2 

4 

.50 

Sub-total 

13 

27 

25 

51 

.03* 

Total 

48 

100 

49 

100 

- 


Note: * p<. 05. 


Social presence. There were no significant differences between iterations in terms of 
social presence (Table 7), although the students in Iteration 2 had a higher frequency of social 
presence. In each iteration, the social presence was largely focused in three areas: compliments, 
directing a comment to someone specific, and emotional expression. A qualitative difference 
among the iterations was that the students in Iteration 1 often expressed concern over doing the 
assignment correctly, while many of the students in Iteration 2 complimented fellow students’ 
ideas as seen in the quote below. 

Iteration 1: 

#3, everything looks great only thing I would say is check with the professor to make 
sure its [sic] ok to number them. I was a bit lost of this because I remember she said no 
bullets. Are numbers considered bullets? I don’t even want to chance it myself. 
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Iteration 2: 

I value any input from my group members. Let me know what I need to work on. 


Table 7 


Differences in Categories within Social Presence 


Element 

Iteration 1 


Iteration 2 


P 


Frequency 

% 

Frequency 

% 


Emotional Expression 

3 

14 

4 

13 

.72 

Open Communication 

17 

81 

25 

81 

.18 

Group Cohesion 

1 

5 

2 

6 

1 

Total 

21 

100 

31 

10 

- 


Teaching presence. There were no significant differences between iterations in terms of 
teaching presence as seen in Table 8. In each iteration, the teaching presence was focused on 
facilitating discourse. This was primarily accomplished by prompting others to generate new 
ideas as a reflection on the business model. Posts were similar across iterations. 


Table 8 

Differences in Categories within Teaching Presence 


Element 

Iteration 1 


Iteration 2 


p 


Frequency 

% 

Frequency 

% 


Instructional Design & 
Organization 

0 

0 

0 

0 

- 

Facilitating Discourse 

8 

100 

9 

100 

1 

Direct Instruction 

0 

0 

0 

0 

- 

Total 

8 

100 

9 

100 

- 


Discussion 

The current study examined the use of discussion protocols in large classes to engage 
students and facilitate discussions. In the following section, the theoretical and practical 
implications of the research findings are discussed. 
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Community of Inquiry 

One unexplored area of research is the use of protocol-based discussions in large 
enrollment courses to improve students’ perception of the Col. After making several 
enhancements (e.g., simplifying the instructions, adding example posts, providing a rubric) in the 
second design iteration of protocol-based discussions, the research team noted changes 
associated with the cognitive presence, social presence, and teaching presence. 

Cognitive presence. After the enhancements were made to the discussion-based 
protocols in Iteration 2, students perceived a significantly higher level of cognitive presence. 
This may be the result of the new perspectives gained from increased feedback received from 
peers. In closely examining one group’s discussion posts from each iteration, the researchers 
also noted a statistically significantly higher level of group cognition in discussion posts in the 
second iteration. This increased group interaction may have resulted from the protocol-based 
discussions becoming more structured with clearer directions and expectations set for what 
students were supposed to do. This result corroborates an earlier study that found that group 
cognition increased with a protocol-based discussion over non-protocol-based discussion 
(Zydney et al., 2012). It may be that increasing the structure of protocol-based 
discussions further improves upon group cognition; however, additional research would 
be needed to confirm this idea. 

Social presence. Similar to what was seen with cognitive presence, the enhancements to 
the discussion protocols in Iteration 2 significantly improved students’ perception of social 
presence in the discussions. Although students’ ratings of social presence improved in the 
second iteration, other data sources showed similarities between the two iterations. For example, 
the analysis of the open-ended survey results revealed a similar theme of community and 
collaboration across both iterations, with students noting feelings of community and belonging. 
Moreover, the analysis of the small group discussions revealed no difference pertaining to social 
presence between the two iterations. This lack of statistical significance may be the result of the 
fact that both design iterations included smaller sub-group discussions that enabled students to 
interact with one another more. (Kim, 2013). 

Teaching presence. Students’ perception of teaching presence also significantly 
improved after enhancements were made to the protocol-based discussions in Iteration 2. This 
difference may be the result of the increased peer feedback noted by students who received the 
second iteration of the discussion protocols. Although students’ ratings of teaching presence 
improved in Iteration 2, examination of the group discussions revealed no statistical difference 
between levels of teaching presence in students’ posts, regardless of which iteration they 
received. This was illustrated by similar numbers of posts from students helping to facilitate 
discussion by encouraging feedback from one another. This lack of statistical significance in 
levels of teaching presence in the discussion posts may be the result of the fact that, regardless of 
the enhancements made, the protocol itself is designed to enable students to facilitate themselves 
in the discussion. This result was also noted in an earlier study that found that the “protocol 
helped the students facilitate themselves and empowered them to design the discussion” (Zydney 
et al., 2012, p. 85). 
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Practical Implications 

The findings of this study generate important implications for facilitating meaningful 
online discussions in large classes. Most importantly, the findings of this study suggest that 
implementing protocol-based discussions is a plausible teaching strategy in classes with large 
enrollments. Students could achieve effective communication and interactions within small 
groups of approximately 10 members in large classes, and the optimal group size is still an 
interesting area for future research. Students in this study enjoyed the small communities, felt a 
sense of connection to classmates, and benefited from peer learning that would not be achievable 
otherwise in large-size classes. Protocols help keep the large community on the same page; 
everyone knows what is expected, including the instructor. 

It is essential to closely attend to the “protocol” nature of the discussion prompt. For 
larger classes, especially, the prompt needs to be explicit, with due dates being clear, and 
examples given. As the research team found, the second design iteration resulted in higher 
perceptions of community and more even distribution of the presences. Being explicit is 
extremely important in very large classes to prevent mass confusion. It is important to explain 
the purpose of the protocol, what to contribute, how to provide feedback to peers, and the 
timeline. In addition, the very nature of the discussion set-up is important. Including the original 
post and peer feedback in one discussion forum enables smooth communication between peers; 
this communication was less frequent in the first iteration when parts A and B were separated. 

It appears that, in large undergraduate classes, students prefer scaffolding with authentic 
examples and simplified, but clear instruction. Although both cohorts felt that they received clear 
instruction for the discussion activity, Iteration 2 was particularly successful due to added 
features such as examples, grading rubrics, and due date reminders. It was surprising that 
students in Iteration 2 were more satisfied with the quality of the feedback, given that more 
detailed direction on how to give peer feedback was provided to students in Iteration 1. The 
design enhancements in the second iteration resulted in fewer concerns over the “correctness” of 
the assigmnent and reassured the students. There was also less concern about when the 
discussion parts were due. This finding suggests a “less is more” instructional design approach; 
instead of providing a large amount of clarified instruction, it may be advantageous to offer the 
most concise instruction possible along with an example. 

It was also surprising that even after adding due date reminders in the course calendar, 
there was still feedback from students in Iteration 2 about needing more reminders of the 
multiple due dates for the protocol-based discussion. This points to the need for more student 
support in the use of the learning management system (LMS). Students need to not only be able 
to submit assigmnents and check grades, but also to skillfully leverage the features of the LMS 
(such as the course calendar) to support their own learning. Therefore, student training emerges 
as an issue for further pursuit. 
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Limitations and Future Research 


There are several limitations to this study. One major limitation is the convenience 
sample selection. Even though this study had a large sample size over two iterations, the data 
only included undergraduate students in one business class at one southeastern university in the 
United States. In future studies, the research team will include classes of various disciplines and 
school enviromnents to assess whether the current findings would generalize to varied contexts 
or samples. Moreover, the researchers only randomly selected one group from each iteration for 
discussion analyses, which limits the sample size of these quantitative analyses. More 
studentgroups should be analyzed to see if the results are consistent in future 
studies. 


In addition, future research will use controlled experimental studies to measure the effect 
of protocol-based discussion on students’ overall learning. Our study results showed a 
statistically significant group cognition but the effect was confounded by other factors, such as 
students’ prior knowledge and the increase in assigned grade to protocol-based discussions in 
Iteration 2. The research team plans to implement pre- and post-examinations for future studies 
to control for individual differences between classes. Also, the effectiveness of modeling in 
protocol-based discussion is one that warrants further research. 

In summary, this study provides some evidence that the protocol-based discussion 
activity can engage students and enhance their cognitive presence in large classes. More research 
is needed in large online classes to further examine the effects of the discussion strategies on 
students’ learning. The researchers hope that this research will be useful to educational 
practitioners and researchers as they continue to investigate and build a knowledge base of 
teaching strategies for large online classes, especially engagement strategies to build personal 
learning communities without having to place heavy logistical burdens on instructors. 
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Appendix A: Iteration 2 Protocol 

For this group discussion posting, I would like you to use a specific structure called a “Tuning” 
protocol. The Tuning process allows for reflection and meaningful feedback about your 
Business Model, and will likely result in your creating a better business model, and earning a 
better grade, too. This structure assumes that you want to improve your business model, and that 
your Bazinga Circle members will deliver thoughtful and substantive feedback. Think of it like 
tuning up a car or an orchestra. Your communal goal is to help each other make A’s on this 
assigmnent, so do this early in the week so you can use their feedback to improve your success 
on your assignment. 

There are three parts to this discussion structure: Parts A, B, and C. 

Part A: Due Week 8 

Begin your post with the words “PART A”. Begin with a one- or two-sentence description of 
your business to remind everyone what your concept is. Share the elements of your emerging 
Business Model. Then ask your Circle members to review your submission and give you 
feedback about something specific in your Business Model you believe could be improved. (For 
example, perhaps you want their feedback on your Revenue Streams or on Validation and 
Pivoting.) 

For Example: (example provided here by the instructor) 

Part B: Due Week 10 

Choose one group member’s Business Model and provide feedback to him or her. Each member 
should give and get feedback from at least one member, so reply to someone who hasn’t received 
any feedback yet. Your communal goal is to help each other make A’s on this assignment. 

Begin your post with the words “Part B”. Then provide feedback that will help your group 
member to improve their submission. 
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For Example: (example provided here by the instructor) 


Part C {to be submitted with the Business Model assignment ): Reflect on what happened as a 
result of this structured discussion. What ideas did your group members provide that helped you 
to improve your Business Model? Who had the best one; why was it better than the others? 

Appendix B: Survey Instrument 


Teaching Presence 

Design & Organization 

1. The instructor clearly communicated important course topics. 

2. The instructor clearly communicated important course goals. 

3. The instructor provided clear instructions on how to participate in course learning activities. 

4. The instructor clearly communicated important due dates/time frames for learning activities. 

Facilitation 

5. The instructor was helpful in identifying areas of agreement and disagreement on course topics that 
helped me to leam. 

6. The instructor was helpful in guiding the class towards understanding course topics in a way that helped 
me clarify my thinking. 

7. The instructor helped to keep course participants engaged and participating in productive dialogue. 

8. The instructor helped keep the course participants on task in a way that helped me to learn. 

9. The instructor encouraged course participants to explore new concepts in this course. 

10. Instructor actions reinforced the development of a sense of community among course participants. 

Direct Instruction 

11. The instructor helped to focus discussion on relevant issues in a way that helped me to leam. 

12. The instructor provided feedback that helped me understand my strengths and weaknesses. 

13. The instructor provided feedback in a timely fashion. 

Social Presence 

Affective expression 

14. Getting to know other course participants gave me a sense of belonging in the course. 

15.1 was able to fonn distinct impressions of some course participants. 

16. Online or web-based communication is an excellent medium for social interaction. 

Open communication 

17.1 felt comfortable conversing through the online medium. 

18.1 felt comfortable participating in the course discussions. 

19.1 felt comfortable interacting with other course participants. 

Group cohesion 

20.1 felt comfortable disagreeing with other course participants while still maintaining a sense of trust. 

21.1 felt that my point of view was acknowledged by other course participants. 

22. Online discussions help me to develop a sense of collaboration. 
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Cognitive Presence 

Triggering event 

23. Problems posed increased my interest in course issues. 

24. Course activities piqued my curiosity. 

25.1 felt motivated to explore content related questions. 

Exploration 

26.1 utilized a variety of infonnation sources to explore problems posed in this course. 

27. Brainstorming and finding relevant infonnation helped me resolve content related questions. 

28. Online discussions were valuable in helping me appreciate different perspectives. 

Integration 

29. Combining new infonnation helped me answer questions raised in course activities. 

30. Learning activities helped me construct explanations/solutions. 

31. Reflection on course content and discussions helped me understand fundamental concepts in 
class. 

Resolution 

32.1 can describe ways to test and apply the knowledge created in this course. 

33.1 have developed solutions to course problems that can be applied in practice. 

34.1 can apply the knowledge created in this course to my work or other non-class related activities. 

5 point Likert-type scale 

1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree 
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